NIC-Based Reduction in Myrinet Clusters: Is It Beneficial?

Reduction-to-one and reduction-to-all operations are common operations in parallel and distributed systems. These operations are collective operations which can involve many processes. It is therefore important to make these operations fast and efficient. Some modern network interface controllers (NICs) for system area networks (SANs) have programmable processors which can be used to offload protocol processing from the host processor. In this paper we investigate the use of the NIC processor to improve the performance of reduction operations. We implemented a NIC-based reduction-to-one operation which can perform integer and floating point operations, and evaluated our implementation. Our evaluation shows that the NIC-based operation performs better than the traditional host-based approach with up to a 1.19 factor of improvement. We also see that using NIC-based reduction can reduce host CPU utilization by a factor of improvement of 2.7, and can reduce the effects of process skew by a factor of improvement of up to 4.5.